This invention relates to digital imaging, and more particularly to edge sharpening during conversion from RGB to YUV formats.
Rapid decrease in the cost of semiconductor devices has significantly increased the amount and complexity of calculations that relatively cheap processors can perform. Compute-intensive applications such as digital imaging can be migrated from the super-computer to the hand-held consumer device. Indeed, some type of digital imaging is widely deployed in stand-alone digital cameras and camcorders.
Devices such as cell phones and personal digital assistants (PDA's) sometimes include digital imaging components, especially as digital imaging is improved and cost-reduced. Digital-camera images are downloaded and stored on personal computers or wirelessly transmitted after capture from an integrated device. Video as well as still images can be captured, depending on the kind of digital camera.
FIG. 1 is a block diagram of a typical digital camera. Light is focused through a lens and directed toward sensor 12, which can be a charge-coupled device (CCD) array or a complementary metal-oxide-semiconductor (CMOS) sensor array. The light falling on the array generates electrical currents, which are amplified by analog amp 14 before being converted from analog to digital values by A/D converter 16. An 8-, 9-, or 10-bit mono-color pixel is output to interpolator 10. These mono-color pixels are in a Bayer-pattern as shown in FIG. 2. Each pixel is either a red, a blue, or a green intensity value. Each pixel has only one of the 3 color components rather than all 3 components per pixel.
The R, G, or B digital values in the Bayer (or Beyer) pattern are processed by interpolator 10 to interpolate missing pixels in the Bayer pattern. The filled-in pixels each have all 3 color components—R, G, and B. The filled-in RGB pixels can then be displayed on display 19.
Processor 15 converts the filled-in pattern of RGB pixels from interpolator 10 to luminance-chrominance YUV pixels. YUV pixels often have a 4:4:4 format, with 8 bits for each of 2 colors and for the luminance. However, other YUV formats such as 4:2:0 may be required by compressor 18, so second converter 11 may be required to convert 4:4:4 YUV pixels to 4:2:2 format, while third converter 11′ converts 4:2:2 YUV pixels to 4:2:0 format. The converted YUV pixels can then be compressed by compressor 18 and stored on disk 17 or on a solid-state memory.
Sensor 12 detects red, blue and green colors. However, each point in the array of sensor 12 can detect only one of the three primary colors. Rather than outputting an RGB pixel, sensor 12 can output only a single-color pixel at any given time. For example, a line of pixels output by sensor 12 might have a red pixel followed by a green pixel. Another line might have alternating green and blue pixels.
Each pixel represents the intensity of one of the primary colors at a point in the sensor array. Thus a red pixel indicates the intensity of red light at a point, while a neighboring green pixel indicates the intensity of green light at the next point in the sensor array. Each pixel thus contains only one-third of the total color information.
The remaining color information is obtained by interpolation. The green intensity of a red pixel is calculated by averaging the green intensities of neighboring green pixels. The blue intensity for that red pixel is calculated by averaging or interpolating the nearest blue pixels. Interpolator 10 performs this color interpolation, calculating the missing primary-color intensities for each pixel location.
The electrical currents produced by the different primary colors can vary, depending on the sensor used and the wavelength and energy of the light photons. An adjustment known as a white-balance is often performed before interpolator 10, either on analog or digital values. Each primary color can be multiplied by a different gain to better balance the colors. Compensation can also be made for different lighting conditions, increasing all primary colors for dark pictures or decreasing all colors for bright pictures (overexposure).
Bayer Pattern FIG. 2
FIG. 2 shows an image captured by a sensor that generates single-color pixels in a Bayer pattern. The example shows an 800×600 frame or image for display in the common super-VGA resolution. A total of 600 lines are captured by the sensor, with 800 pixels per line.
Personal computers and many other devices display full-color pixels that have all three primary-color intensities (RGB). In contrast, the sensor in a digital camera can detect only one of the three primary colors for each point in the 800×600 sensor array. Detectors for green are alternated with red detectors in the first line, while green detectors are alternated with blue detectors in the second line.
The first horizontal line and each odd line have alternating red and green detectors, so pixels output from these odd lines are in a G-R-G-R-G-R-G sequence. The second horizontal line and each even line have alternating green and blue detectors, so pixels output from these even lines are in a B-G-B-G-B-G-B sequence.
Half of the pixels are green pixels, while one-quarter of the pixels are read and the last quarter are blue. The green pixels form a checkerboard pattern, with blue and red pixels surrounded by green pixels. Since the human eye is more sensitive to green, the Bayer pattern has more green pixels than red or blue.
The green intensity for a red pixel location can be interpolated by averaging the four green pixels that surround the red pixel. For example, the green intensity for red pixel at location (3,2) is the sum of green pixels (3,1), (3,3), (2,2), and (4,2), divided by four. Likewise, the green intensity for a blue pixel location can be interpolated by averaging the four surrounding green pixels. For blue pixel (2,3), the interpolated green intensity is the sum of green pixels (2,2), (2,4), (1,3), and (3,3), divided by four.
The red and blue values for a green pixel location can also be calculated from the 2 red and 2 blue pixels that surround each green pixel. For green pixel (2,2), the interpolated red value is the average of red pixels (1,2) and (3,2) above and below the green pixel, while the interpolated blue value is the average of blue pixels (2,1) and (2,3) to the right and left of the green pixel.
Color interpolation, analog-to-digital conversion, and resolution of the sensor may cause images to be somewhat fuzzy. In particular, edges of objects in the image may not appear to the eye to be as sharp as when the object is viewed directly by the eye.
Image processing can be used to enhance the sharpness of edges in the image. Object edges often have rapid changes in color or brightness as pixels change from being captured from the object to being captured from the background. This rapid color or brightness change is typical of an edge and can be enhanced by image processing. For example, when a gradient or change in color or brightness is detected by an image processor, the gradient can be increased to produce a larger, sharper gradient. The sharper gradient increases the perception of sharpness of object edges.
While digital-camera processors are useful, cost reduction is desirable since digital cameras are price-sensitive consumer devices. Digital cameras may be a component of an integrated device such as a cell phone or PDA. Such integrated devices may use image compression to allow pictures to be compactly transmitted over wireless networks. Compression may require one or more format conversions. Edge enhancement may be desirable prior to compression. These format conversions and edge enhancements add to complexity and cost of the device.
What is desired is a digital-image processor that does not require interpolation on RGB pixels in the Bayer pattern. It is desired to combine color interpolation of Bayer-pattern pixels with format conversion to YUV. It is desired to use a single step conversion that both interpolates and generates YUV formats from a Bayer-pattern input. It is further desired to perform edge enhancement in an integrated manner with the conversion processing.